Real Time Segmentation and Tracking of Face and Hands in VR Applications

We describe a robust real-time 3D tracking system of the extreme limbs of the upper human body, i.e., the hands and the face. The goal of the system is that it can be used as a perceptual interface for virtual reality activities in a workbench environment. The whole system includes an input capture and calibration module, a real time color segmentation module, a data association and tracking module and finally a visualization VRML and H-ANIM procedure. The results of our probabilistically skin-color segmentation are skincolor blobs. Then, for each frame of the sequence our algorithm labels the blobs’ pixels using a set of object state hypothesis. This set of hypothesis is built from the results of previous frames. The 2D tracking results are used for the 3D reconstruction of limbs position in order to obtain the H-ANIM visualization results. Several results are presented to show the algorithm performance.